Image processing device and image processing method

ABSTRACT

An output value calculating unit  5  performs weighting arithmetic operations on a plurality of filter values calculated by a filter value calculating unit  4  by using a feature quantity detected by a feature quantity detecting unit  1 , a directional correlation value calculated by a directional correlation value calculating unit  2  and an amplitude level calculated by an amplitude level calculating unit  3 , and multiplies a result of the weighting arithmetic operations by a sensitization magnification to create pixel signals of a correctly exposed image.

FIELD OF THE INVENTION

The present invention relates to an image processing device for and animage processing method of acquiring a sensitized image having a highresolution.

BACKGROUND OF THE INVENTION

A digital still camera equipped with an image sensor, such as a CCD,typically carries out high-sensitivity image capturing with a raisedimage signal level by performing a long exposure or electricallyamplifying an image signal outputted from the image sensor when makingan appropriate exposure under conditions of low illumination, such as atnighttime, without using any flush or the like, for example.

However, because a camera movement or a movement of the object to bephotographed easily occurs during which such a digital still cameraperforms a long exposure, the image capturing environment is subjectedto a constraint in many cases, such as a constraint that the digitalstill camera needs to be fixed to a tripod to shoot the object to bephotographed or a constraint that any moving object, such as a human, isnot chosen as the object to be photographed.

A further problem is that because the noise mixed into the image signalin the analog circuit of the camera is amplified, together with theimage information, at a high amplification factor when carrying out highsensitivity image capturing, the S/N ratio of the image degrades.

To solve this problem, the following patent reference 1discloses animage processing method of using information about an image capturedwith a relatively low degree of sensitivity among two images capturedwith different degrees of sensitivity and different exposure times toremove the noise of the image captured with a relatively high degree ofsensitivity.

However, because a camera movement and a movement of the object to bephotographed cannot be prevented from occurring while the two images arecaptured even when this image processing method is used, constraints asmentioned above are still imposed on the image capturing environment.

Furthermore, although a method of using a digital pixel mixing processof adding the signal levels of neighborhood image elements within animage acquired through one image capturing without using a plurality ofimages to cancel random noise while raising the signal level of theobject to be photographed is also used, a problem with this method isthat the resolution of the object to be photographed decreasesremarkably.

RELATED ART DOCUMENT Patent reference

-   Patent reference 1: JP,2007-312284, A (paragraph number [0009] and    FIG. 1)

SUMMARY OF THE INVENTION

Because the conventional image processing method is configured asmentioned above, while the noise of an image captured with a relativelyhigh degree of sensitivity can be removed, a camera movement and amovement of the object to be photographed cannot be prevented fromoccurring while two images are captured. A problem is therefore thatmany constraints are imposed on the image capturing environment.

The present invention is made in order to solve the above-mentionedproblem, and it is therefore an object of the present invention toprovide an image processing device and an image processing method whichcan provide a low-noise correctly exposed image with a high resolutionwithout constraints on the image capturing environment.

An image processing device in accordance with the present inventionincludes: a feature quantity detecting unit for detecting a featurequantity of an object to be photographed which exists around an aimedpixel in a two-dimensional image; a directional correlation valuecalculating unit for calculating a directional correlation value of theobject to be photographed which exists around the aimed pixel from thefeature quantity detected by the feature quantity detecting unit; and afilter value calculating unit for referring to a pixel signal of theaimed pixel and pixel signals of neighboring pixels in a neighborhood ofthe aimed pixel to calculate a plurality of filter values, and acorrectly exposed image creating unit performs weighting arithmeticoperations on the plurality of filter values calculated by the filtervalue calculating unit by using the feature quantity detected by thefeature quantity detecting unit and the directional correlation valuecalculated by the directional correlation value calculating unit, andfor multiplying a result of the above-mentioned weighting arithmeticoperations by a sensitization magnification to create pixel signals of acorrectly exposed image.

In accordance with the present invention, because the image processingdevice is constructed in such a way that it includes the featurequantity detecting unit for detecting a feature quantity of an object tobe photographed which exists around an aimed pixel in a two-dimensionalimage, the directional correlation value calculating unit forcalculating a directional correlation value of the object to bephotographed which exists around the aimed pixel from the featurequantity detected by the feature quantity detecting unit, and the filtervalue calculating unit for referring to a pixel signal of the aimedpixel and pixel signals of neighboring pixels in a neighborhood of theaimed pixel to calculate a plurality of filter values, and the correctlyexposed image creating unit performs weighting arithmetic operations onthe plurality of filter values calculated by the filter valuecalculating unit by using the feature quantity detected by the featurequantity detecting unit and the directional correlation value calculatedby the directional correlation value calculating unit, and formultiplying a result of the above-mentioned weighting arithmeticoperations by a sensitization magnification to create pixel signals of acorrectly exposed image, the present invention offers an advantage ofbeing able to provide a high-resolution low-noise correctly exposedimage without being restricted by the image capturing environment.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing an image processing device inaccordance with Embodiment 1 of the present invention;

FIG. 2 is a flow chart showing processing carried out by the imageprocessing device in accordance with Embodiment 1 of the presentinvention;

FIG. 3 is a schematic diagram showing a light receiving element groupwhich is a primary color Bayer-type single plate image sensor usedtypically for a digital still camera;

FIG. 4 is an explanatory drawing showing a relationship among imageelement positions at the time of carrying out a feature quantitydetection;

FIG. 5 is an explanatory drawing showing a filter arithmetic operationusing an aimed pixel and horizontal three pixels;

FIG. 6 is an explanatory drawing showing a filter arithmetic operationusing the aimed pixel and vertical three pixels;

FIG. 7 is an explanatory drawing showing a filter arithmetic operationusing nine pixels including the aimed pixel;

FIG. 8 is an explanatory drawing schematically showing an arrangement oflight receiving elements and color filters of a representativecomplementary color system image sensor;

FIG. 9 is a schematic diagram showing a distribution of luminancesignals Y of a brightness color difference system signal into which thetwo-dimensional image of FIG. 8 is converted;

FIG. 10 is an explanatory drawing showing the positions of referencepixels corresponding to filter arithmetic operations given by equations(19) and (20);

FIG. 11 is an explanatory drawing showing the positions of referencepixels corresponding to filter arithmetic operations given by equations(21) and (22);

FIG. 12 is an explanatory drawing showing the positions of referencepixels corresponding to filter arithmetic operations given by equations(23) and (24);

FIG. 13 is an explanatory drawing showing a distribution of availableneighboring pixels; and

FIG. 14 is an explanatory drawing showing an example of settings of athreshold TH1 and a threshold TH2.

EMBODIMENTS OF THE INVENTION

Hereafter, in order to explain this invention in greater detail, thepreferred embodiments of the present invention will be described withreference to the accompanying drawings.

Embodiment 1.

FIG. 1 is a block diagram showing an image processing device inaccordance with Embodiment 1 of the present invention.

In FIG. 1, a feature quantity detecting unit 1 carries out a process of,when receiving a two-dimensional image, detecting a feature quantity ofan object to be photographed which exists around an aimed pixel in thetwo-dimensional image. The feature quantity detecting unit 1 constructsa feature quantity detecting unit.

A directional correlation value calculating unit 2 carries out a processof calculating a directional correlation value of the object to bephotographed which exists around the aimed pixel from the featurequantity detected by the feature quantity detecting unit 1. Thedirectional correlation value calculating unit 2 constructs adirectional correlation value calculating unit.

An amplitude level calculating unit 3 carries out a process ofcalculating an amplitude level of the aimed pixel with respect toneighboring pixels (pixels existing around the aimed pixel) from thefeature quantity detected by the feature quantity detecting unit 1. Theamplitude level calculating unit 3 constructs an amplitude levelcalculating unit.

A filter value calculating unit 4 carries out a process of referring toa pixel signal of the aimed pixel and pixel signals of the neighboringpixels to calculate a plurality of filter values (e.g., a low passfilter value in a one-dimensional direction including the aimed pixel, amedian filter value in a one-dimensional direction including the aimedpixel, a two-dimensional low pass filter value including the aimedpixel, and a two-dimensional median filter value including the aimedpixel). The filter value calculating unit 4 constructs a filter valuecalculating unit.

An output value calculating unit 5 carries out a process of performingweighting arithmetic operations on the plurality of filter valuescalculated by the filter value calculating unit 4 by using the featurequantity detected by the feature quantity detecting unit 1, thedirectional correlation value calculated by the directional correlationvalue calculating unit 2, and the amplitude level calculated by theamplitude level calculating unit 3, and multiplying the final result ofthe weighting arithmetic operations by a sensitization magnification tocreate pixel signals of a correctly exposed image. The output valuecalculating unit 5 constructs a correctly exposed image creating unit.

FIG. 2 is a flow chart showing processing carried out by the imageprocessing device in accordance with Embodiment 1 of the presentinvention.

FIG. 3 is a schematic diagram showing a light receiving element groupwhich is a primary color Bayer-type single plate image sensor usedtypically for a digital still camera.

In FIG. 3, each square shows one light receiving element, and the lightreceiving element group arranged in two dimensions carries outphotoelectric conversion to capture a two-dimensional image of theobject to be photographed.

A unit which uses a single set of this type of two-dimensional lightreceiving element group is called a single plate image sensor. In orderto be able to capture a full color image by using the singletwo-dimensional light receiving element group, the singletwo-dimensional light receiving element group has color filters arrangedrespectively for assigning colors to be photographed to the plurality oflight receiving elements.

In the example of FIG. 3, each light receiving element to which a G(green) color filter is attached photographs only a G color, each lightreceiving element to which a R (red) color filter is attachedphotographs only a R color, and each light receiving element to which aB (blue) color filter is attached photographs only a B color. Thearrangement in which the color filters are arranged as shown in FIG. 3is called a Bayer array, and is used typically.

Next, the operation of the image processing device will be explained.

In this Embodiment 1,an image is captured by using the primary colorBayer-type single plate image sensor shown in FIG. 3.

In the following explanation, it is assumed that each processing unitcarries out a process with a processing window which consists of 5pixels in horizontal direction ×5 pixels in vertical direction around anaimed pixel.

When receiving a two-dimensional image captured, the feature quantitydetecting unit 1 detects a feature quantity of an object to bephotographed which exists around an aimed pixel in the two-dimensionalimage first (step ST1).

Hereafter, the detecting process of detecting a feature quantity whichis performed by the feature quantity detecting unit 1 will be explainedconcretely.

FIG. 4 is an explanatory drawing showing a relationship among imageelement positions at the time of carrying out the feature quantitydetection.

In this Embodiment 1,it is assumed that as to G signals, G of a GR line(referred to as “Gr” from here on) and G of a GB line (referred to as“Gb” from here on) are handled as signals of different colors.

In FIG. 4, “P” shows a pixel having an imaging signal of either one ofthe R color, the Gr color, the Gb color and the B color.

Furthermore, “P22” is the aimed pixel which is the target forprocessing, 9 pixels P00, . . . , and P44 including P22 have the samecolor as P22 in the 5×5 window.

The feature quantity detecting unit 1 detects a feature quantity of anobject to be photographed which exists around the aimed pixel, asmentioned above. Concretely, as shown in the following equations (1) and(2), the feature quantity detecting unit 1 calculates a horizontalsecondary differential value IntH of the aimed pixel and a verticalsecondary differential value IntV of the aimed pixel as the featurequantity of the object to be photographed.IntH=|P22−P02|+|P22−P42|  (1)IntV=|P22−P20|+|P22−P24|  (2)

For example, in a case in which an edge of the object to be photographedis crossing in a vertical direction when viewed from P22 of FIG. 4, thesecondary differential value IntH has a relatively large value while thesecondary differential value IntV has a relatively small value.

In contrast with this, in a case in which an edge of the object to bephotographed is crossing in a horizontal direction when viewed from P22of FIG. 4, the secondary differential value IntV has a relatively largevalue while the secondary differential value IntH has a relatively smallvalue.

After the feature quantity detecting unit 1 calculates the secondarydifferential values IntH and IntV as the feature quantity, thedirectional correlation value calculating unit 2 calculates adirectional correlation value IntHV of the object to be photographedwhich exists around the aimed pixel (when the aimed pixel has a highcorrelation with either one of a pixel in the horizontal direction andthat in the vertical direction, an index value showing that the aimedpixel has a high correlation with either one of a pixel in thehorizontal direction and that in the vertical direction) by using thesecondary differential values IntH and IntV (step ST2), as shown in thefollowing equation (3).IntHV=IntH−IntV|  (3)

In the equation (3), because the difference between the secondarydifferential value IntH and the secondary differential value IntVbecomes large when the aimed pixel has a high correlation with eitherone of a pixel in the horizontal direction and that in the verticaldirection, the directional correlation value IntHV becomes a relativelylarge value.

In contrast with this, when the aimed pixel has the degree ofdirectional correlation with each of both a pixel in the horizontaldirection and that in the vertical direction (e.g., when the object tobe photographed is positioned in a flat portion which is located not inthe vicinity of an edge or when the object to be photographed is at thetop of an uneven portion), the directional correlation value IntHVbecomes a relatively small value.

After the feature quantity detecting unit 1 calculates the secondarydifferential values IntH and IntV as the feature quantity, the amplitudelevel calculating unit 3 calculates an amplitude level DifHV of theaimed pixel with respect to the neighboring pixels (an amplitude indexvalue of a signal level showing what degree of unevenness the aimedpixel has with respect to the neighboring pixels) by using the secondarydifferential values IntH and IntV (step ST3), as shown in the followingequation (4).DifHV=IntH+IntV  (4)

In the equation (4), because the secondary differential values IntH andIntV are both relatively small when the aimed pixel is positioned in aflat portion which is not located in the vicinity of any edge, theamplitude level DifHV becomes a small value.

In contrast with this, because the secondary differential values IntHand IntV are both relatively large when the aimed pixel is at the top ofan uneven portion, the amplitude level DifHV becomes a large value.

The filter value calculating unit 4 refers to the pixel signal of theaimed pixel and the pixel signals of the neighboring pixels to calculatea plurality of filter values (e.g., a low pass filter value in aone-dimensional direction including the aimed pixel, a median filtervalue in a one-dimensional direction including the aimed pixel, atwo-dimensional low pass filter value including the aimed pixel, and atwo-dimensional median filter value including the aimed pixel) (stepST4).

More specifically, the filter value calculating unit 4 carries outfilter arithmetic operations using the horizontal 3 pixels shown in FIG.5 including the aimed pixel, filter arithmetic operations using thevertical 3 pixels shown in FIG. 6 including the aimed pixel, and filterarithmetic operations using the 9 pixels shown in FIG. 7 including theaimed pixel.

In performing the filter arithmetic operations in the horizontaldirection of FIG. 5 among these filter arithmetic operations, the filtervalue calculating unit calculates a low pass filter value lpf000 byusing the following equation (5), and also calculates a 3-pixel medianfilter value med000 by using the following equation (6).lpf000=(P22+P02+P42)/3  (5)med000=[the median value of P22, P02, and P42]  (6)

Furthermore, in performing the filter arithmetic operations in thevertical direction of FIG. 6 among the filter arithmetic operations, thefilter value calculating unit calculates a low pass filter value lpf090by using the following equation (7), and also calculates a 3-pixelmedian filter value med090 by using the following equation (8).lpf090=(P22+P20+P24)/3  (7)med090=[the median value of P22, P20, and P24]  (8)

Furthermore, in performing the filter arithmetic operations shown inFIG. 7 among the filter arithmetic operations, the filter valuecalculating unit calculates a low pass filter value Alpf by using thefollowing equation (9) and also calculates a median filter value Amed byusing the following equation (10).Alpf=(P22+P00+P20+P40+P02+P42+P04+P24+P44)/9   (9)Amed=[the median value of P22,P00,P20,P40,P02,P42, P04,P24,andP44]  (10)

The output value calculating unit 5 performs weighting arithmeticoperations on the plurality of filter values calculated by the filtervalue calculating unit 4 by using the secondary differential values IntHand IntV, which are the feature quantity detected by the featurequantity detecting unit 1, the directional correlation value IntHVcalculated by the directional correlation value calculating unit 2, andthe amplitude level DifHV calculated by the amplitude level calculatingunit 3 and multiplies the final result of the weighting arithmeticoperations by a sensitization magnification DGain to create pixelsignals Out of a correctly exposed image (step ST5).

Hereafter, the process of creating the pixel signals Out of thecorrectly exposed image which is carried out by the output valuecalculating unit 5 will be explained concretely.

First, the output value calculating unit 5 performs weighting additionon the low pass filter value and the median filter value for eachdirection by using amplitude level DifHV calculated by the amplitudelevel calculating unit 3.

The following equation (11) shows the weighting addition performed onthe horizontal low pass filter value and the horizontal median filtervalue which correspond to FIG. 5, and the following equation (12) showsthe weighting addition performed on the vertical low pass filter valueand the vertical median filter value which correspond to FIG. 6.

Furthermore, the following equation (13) shows the weighting additionperformed on the low pass filter value and the median filter value whichcorrespond to FIG. 7.

In the equations (11) to (13), Diflimit shows a maximum which theamplitude level DifHV can have, and is a fixed value which is set up inadvance.

Furthermore, it is assumed that the amplitude level DifHV is subjectedto a limiting process according to the Diflimit value in advance.dlpf000={DifHV×med000+(Diflimit−DifHV)×lpf000}/Diflimit  (11)dlpf090={DifHV×med090+(Diflimit−DifHV)×lpf090}/Diflimit  (12)Alpf2={DifHV×Amed+(Diflimit−DifHV)×Alpf}/Diflimit  (13)

By performing the calculations according to the equations (11) to (13),when the aimed pixel has a larger amplitude than the neighboring pixels,the output value calculating unit 5 can increase the weight of themedian filter value. Therefore, when the aimed pixel is isolated pointnoise, the output value calculating unit can easily remove the isolatedpoint noise.

Furthermore, when the aimed pixel has the same amplitude as theneighboring pixels and its amplitude is small, the output valuecalculating unit can increase the weight of the low pass filter value,and can therefore enhance an image smoothing effect.

Next, the output value calculating unit 5 performs weighting addition onthe filter value in each direction according to the secondarydifferential values IntH and IntV each of which is a feature quantitydetected by the feature quantity detecting unit 1, as shown in thefollowing equation (14).Dlpf=(IntV×dlpf000+IntH×dlpf090)/DifHV  (14)

When the aimed pixel is on a vertical edge, the weight of the filtervalue dlpf090 in the vertical direction becomes large in Dlpf shown bythe equation (14) because the secondary differential value IntH is largeand the secondary differential value IntV is small.

In contrast, when the aimed pixel is on a horizontal edge, the weight ofthe filter value dlpf000 in the horizontal direction becomes large inDlpf because the secondary differential value IntV is large and thesecondary differential value IntH is small.

Next, the output value calculating unit 5 carries out weighting additionof Dlpf, to which a heavier weight has been assigned with respect to thehorizontal or vertical direction, and Alpf2 which is uniform in theregion by using the directional correlation value IntHV calculated bythe directional correlation value calculating unit 2, as shown in thefollowing equation (15), to calculate the pixel signals Out of thecorrectly exposed image which are a final output value.Out={IntHV×Dlpf+(Intlimit−IntHV)×Alpf2}×DGain/Intlimit  (15)

In this equation (15), Intlimit shows a maximum which the directionalcorrelation value IntHV can have, and is a fixed value which is set upin advance.

Furthermore, it is assumed that the directional correlation value IntHVis subjected to a limiting process according to the Intlimit value inadvance.

In addition, DGain shows the sensitization magnification which is usedto carry out digital sensitization.

It can be seen from the equation (15) that when the directionalcorrelation is low, a heavier weight is assigned to the filter valueAlpf2 which is uniform in the region, whereas when the directionalcorrelation is high, a heavier weight is assigned to the directivityfilter value Dlpf.

As can be seen from the above description, the image processing devicein accordance with this Embodiment 1 is constructed in such away that itincludes the feature quantity detecting unit 1 for detecting a featurequantity of an object to be photographed which exists around an aimedpixel in a two-dimensional image, the directional correlation valuecalculating unit 2 for calculating a directional correlation value ofthe object to be photographed which exists around the aimed pixel fromthe feature quantity detected by the feature quantity detecting unit 1,the amplitude level calculating unit 3 for calculating an amplitudelevel of the aimed pixel with respect to neighboring pixels from thefeature quantity detected by the feature quantity detecting unit 1, andthe filter value calculating unit 4 for calculating a plurality offilter values with reference to a pixel signal of the aimed pixel andpixel signals of the neighboring pixels, and the output valuecalculating unit 5 performs weighting arithmetic operations on theplurality of filter values calculated by the filter value calculatingunit 4 by using the feature quantity detected by the feature quantitydetecting unit 1, the directional correlation value calculated by thedirectional correlation value calculating unit 2, and the amplitudelevel calculated by the amplitude level calculating unit 3, andmultiplies the final result of the weighting arithmetic operations by asensitization magnification to create pixel signals of a correctlyexposed image. Therefore, this embodiment offers an advantage of beingable to provide a high-resolution low-noise correctly exposed imagewithout being restricted by the image capturing environment.

More specifically, because the image processing device in accordancewith this Embodiment 1 carries the sensitization process in a singlescreen, the image processing device becomes possible to carry out thesensitization process without being affected by the influence of acamera movement and a movement of the object to be photographed whichoccurs in the sensitization process using multiple frames.

Furthermore, because the image processing device calculates the pixelsignals of the correctly exposed image by using the filters having anoise reduction effect, such as low pass filters and median filters, theimage processing device can carry out the sensitizing process with a lownoise level compared with that in a case in which an inputted imagehaving an under exposure is signal-amplified by an analog circuit, andthat in a case in which digital data are simply signal-amplified.

Furthermore, because the image processing device detects a featurequantity of an object to be photographed in a small area and calculatesa directional correlation value, and then carries out weighting additionof filter values having directivity according to the directionalcorrelation value, the image processing device can acquire a sensitizedimage having a high resolution compared with that acquired through aconventional pixel mixing process of simply adding the values ofneighboring pixels.

In addition, because the image processing device detects a featurequantity of an object to be photographed in a small area and calculatesthe amplitude level of an aimed pixel, and then carries out weightingaddition of a low pass filter value and a median filter value accordingto the amplitude level, the image processing device can acquire asensitized image having high quality in which fluctuations caused by thenoise of a flat tone portion are suppressed while random noise isremoved from the image.

Furthermore, because the image processing device detects a featurequantity of an object to be photographed in a small area and calculatesvarious weighting factors for weighting addition from the featurequantity, and then carries out weighting addition of the plurality offilter values, the image processing device can vary the filters appliedto the sensitization process seamlessly. Therefore, the image processingdevice has a feature of making it difficult for image qualitydegradation resulting from a sudden change per pixel in the texture ofthe image to occur compared with a case in which the filters applied tothe sensitization process are changed according to the results of thedetection of features from an area.

Embodiment 2.

In above-mentioned Embodiment 1,the example in which a two-dimensionalimage captured by using the primary color system Bayer-type single plateimage sensor shown in FIG. 3 is inputted into the image processingdevice is shown. As an alternative, a two-dimensional image captured byusing another image sensor can be inputted into the image processingdevice.

In a case in which the type of the image sensor applied to the imageprocessing device differs in this way, the arithmetic operationperformed by the feature quantity detecting unit 1 and that performed bythe filter value calculating unit 4 change to some extent.

FIG. 8 is an explanatory drawing schematically showing an arrangement oflight receiving elements and color filters of a representativecomplementary color system image sensor.

In the complementary color system image sensor, complementary colorfilters of Ye (yellow), Mg (magenta), and Cy (cyan) are arranged, and Gcolor filter are further arranged for brightness detection.

While only G exhibits an adequate response to brightness in a primarycolor system sensor, a color filter used for the complementary colorsystem sensor and corresponding to each pixel easily exhibits a responseto brightness, and brightness information can be extracted from allpixels. Therefore, the complementary color system sensor has a featureof providing a captured image having a high resolution.

The image processing device in accordance with this Embodiment 2processes the two-dimensional image which the image processing devicehas captured by using the complementary color system image sensor bychanging a reference pixel position at the time of arithmetic operationsafter converting the two-dimensional image into a brightness colordifference system signal (not shown) in order to process thetwo-dimensional image while taking advantage of the high resolutionperformance of the complementary color system image sensor.

FIG. 9 is a schematic diagram showing a distribution of luminancesignals Y of the brightness color difference system signal into whichthe two-dimensional image shown in FIG. 8 is converted.

The two-dimensional image captured by using the complementary colorsystem image sensor can be converted into the luminance signals Y byusing the following equation (16) with 2 pixels in a horizontaldirection and 2 pixels in a vertical direction being defined as oneunit.Y=Mg+G+Cy+Ye  (16)

Because the image processing device in accordance with above-mentionedEmbodiment 1 carries out the detection of a feature quantity, and thecalculation of filter values by using pixels of the same color, theimage processing device uses alternate pixels in the two-dimensionalimage, as shown in FIGS. 4 to 7. In contrast, because the luminancesignals are distributed over all the pixels of the complementary colorsystem image sensor, as shown in FIG. 9, the image processing device inaccordance with this embodiment differs from that in accordance withabove-mentioned Embodiment 1 in that the image processing device inaccordance with this embodiment performs arithmetic operations by usingpixels Y11 to Y33 arranged close to an aimed pixel Y22.

The equations (1) and (2) are transformed into the following equations(17) and (18) as shown below, for example, and the feature quantitydetecting unit 1 calculates secondary differential values IntH and IntVeach of which is a feature quantity by using the equations (17) and(18).IntH=|Y22−Y12|+Y22−Y32|  (17)IntV=|Y22−Y21|+Y22−Y23|  (18)

The filter value calculating unit 4 calculates low pass filter valuesand median filter values by using the following equations (19) to (24),for example, instead of the equations (5) to (10).

FIG. 10 is an explanatory drawing showing the positions of referencepixels corresponding to filter arithmetic operations given by theequations (19) and (20), and FIG. 11 is an explanatory drawing showingthe positions of reference pixels corresponding to filter arithmeticoperations given by the equations (21) and (22).

Furthermore, FIG. 12 is an explanatory drawing showing the positions ofreference pixels corresponding to filter arithmetic operations given bythe equations (23) and (24).lpf000=(Y22+Y12+Y32)/3  (19)med000=[the median value of Y22, Y12, and Y32]  (20)lpf090=(Y22+Y21+Y23)/3  (21)med090=[the median value of Y22, Y21, and Y23]  (22)Alpf=(Y22+Y11+Y21+Y31+Y12+Y32+Y13+Y23+Y33)/9   (23)Amed=[the median value of Y22,Y11,Y21,Y31,Y12,Y32, Y13,Y23,andY33]  (24)

Because feature quantities of any adjacent pixels not including a pixelof the same color cannot be used in the processing performed on theprimary color system image sensor, a feature quantity of the object tobe photographed cannot be detected appropriately when striped patternsexist at every other pixel of the object to be photographed, forexample. In contrast, as shown in this Embodiment 2, because the imageprocessing device can detect feature quantities of pixels includingadjacent pixels by processing the captured image by using thecomplementary color system image sensor, the image processing device canimplement the pixel mixing process with a higher resolution.

Embodiment 3.

In above-mentioned Embodiment 1,the example in which Gb and Gr areprocessed as different colors is shown. As an alternative, Gb and Gr canbe processed as the same G color.

In this case, available neighboring pixels have a distribution as shownin FIG. 13. Therefore, unlike in the case of FIG. 7, pixels diagonallyadjacent to an aimed pixel can be used. By thus using pixels diagonallyadjacent to an aimed pixel for the detection of feature quantities orthe determination of a directional correlation value, the imageprocessing device can implement processing with a higher resolution.

In above-mentioned Embodiments 1 and 2,the image processing deviceautomatically carries out the weighting addition, as shown in theequation (15), of the filter value Dlpf and the filter value Alpf2according to the directional correlation value IntHV, as shown above.This weighting addition process is only an example, and the imageprocessing device can be alternatively constructed in such a way as tocontrol the process according to a user's choice.

For example, the user can set up a threshold TH1 and a threshold TH2 inadvance (refer to FIG. 14), and, when IntHV<=TH1, the image processingdevice determines the filter value Alpf2 as a final output value Outunconditionally, whereas when IntHV>=TH2, the image processing devicedetermines the filter value Dlpf as the final output value Outunconditionally.

Furthermore, when TH1<IntHV<TH2, the image processing device carries outweighting addition using the directional correlation value IntHVaccording to the equation (15).

As a result, the image processing device provides an advantage ofincreasing the flexibility of adjustment of the image quality accordingto the user's choice.

In above-mentioned Embodiments 1 and 2,the example in which theweighting addition of the filter value Dlpf and the filter value Alpf2is multiplied by the sensitization magnification DGain in the equation(15) is shown. As an alternative, when performing each of the processesof calculating dlpf000 and dlpf090,and Alpf and Alpf2,the imageprocessing device can multiply the weighting addition by DGain beforeperforming the division.

By thus multiplying the weighting addition by the sensitizationmagnification DGain before performing the division in each of theequations (9), (11), (12), and (13), the image processing device caninclude the sensitization magnification DGain in the rounded operationat the time of performing the division. As a result, the imageprocessing device can implement pixel value arithmetic operations with ahigher degree of accuracy.

In above-mentioned Embodiments 1 and 2,the image processing devicecarries out the weighting addition, as shown in the equation (13), ofthe filter value Amed and the filter value Alpf according to theamplitude level DifHV, as shown above. This weighting addition processis only an example, and the image processing device can be alternativelyconstructed in such a way as to always use the filter value Alpf as thefilter value which is uniform in the region.

In this case, although it becomes difficult to reduce random noise, suchas an isolated point, in an area having a low degree of directivity, theimage processing device does not have to carry out the arithmeticoperations according to the equations (10) and (13). Therefore, when theimage processing device is implemented as a circuit, this variant iseffective for reduction of the scale of the circuit.

In above-mentioned Embodiments 1 and 2,the image processing devicecarries out the weighting addition, as shown in the equations (11) and(12), of the median filter value and the low pass filter value accordingto the amplitude level DifHV, as shown above. This weighting additionprocess is only an example, and the image processing device can bealternatively constructed in such a way as to enable the user to preseta parameter corresponding to the amplitude level DifHV as a fixedparameter, for example, and fix the weight assigned to the median filtervalue and that assigned to the low pass filter value.

In this case, the image processing device provides an advantage ofenabling the user to freely adjust the image quality. Furthermore, bycombining with a structure of not carrying out the weighting addition ofthe filter value Amed and the filter value Alpf according to theequation (13), the image processing device does not have to calculatethe amplitude level DifHV (the amplitude level calculating unit 3becomes unnecessary). Therefore, when the image processing device isimplemented as a circuit, this variant is effective for reduction of thescale of the circuit.

In above-mentioned Embodiments 1 and 2,the image processing deviceprocesses a two-dimensional image inputted thereto from the imagesensor, as shown above. However, the present invention is not limited tothis example. As an alternative, the image processing device can beconstructed in such a way as to process only an image in which RGBsignals or brightness color difference system signals are provided foreach pixel.

More specifically, although the image processing device in accordancewith above-mentioned Embodiment 2 carries out the processing aftercalculating luminance signals for all the pixels, the image processingdevice can be alternatively constructed in such a way as to carry outthe same processing assuming that signals to be processes are providedfor all the pixels.

In above-mentioned Embodiments 1 and 2,the feature quantity detectingunit 1 calculates secondary differential values as feature quantities,as shown above. However, the present invention is not limited to thisexample. In the case of a physical quantity showing the shape of anobject to be photographed in an area, the feature quantity detectingunit can calculate primary differential values or a degree of similarityby comparing patterns which are predefined in order to detect ahorizontal edge and a vertical edge with the pixel signal values of thearea, for example. This variant can provide the same advantages.

Furthermore, in above-mentioned Embodiments 1 and 2,the feature quantitydetecting unit 1 detects feature quantities in both the horizontal andvertical directions, as shown above. However, the present invention isnot limited to this example. For example, the feature quantity detectingunit can detect feature quantities in diagonal directions by using thepixels P00, P40, P04, and P44 which are located in diagonal directions,as shown in FIG. 4, with respect to the aimed pixel.

In this case, while the amount of arithmetic operation increases, anedge in a diagonal direction can also be reproduced with a highresolution.

INDUSTRIAL APPLICABILITY

The image processing device and the image processing method inaccordance with the present invention provide an advantage of being ableto acquire a high-resolution low-noise correctly exposed image withoutbeing restricted by the image capturing environment. Therefore, theimage processing device and the image processing method in accordancewith the present invention are suitable for use as an image processingmethod and an image processing device which acquire a sensitized imagehaving a high resolution, respectively, and so on.

The invention claimed is:
 1. An image processing device comprising: afeature quantity detecting unit for detecting a feature quantity of anobject to be photographed which exists around an aimed pixel in atwo-dimensional image; a directional correlation value calculating unitfor calculating a directional correlation value of the object to bephotographed which exists around said aimed pixel from the featurequantity detected by said feature quantity detecting unit; a filtervalue calculating unit for referring to a pixel signal of said aimedpixel and pixel signals of neighboring pixels in a neighborhood of saidaimed pixel to calculate a plurality of filter values; and a correctlyexposed image creating unit for performing weighting arithmeticoperations on the plurality of filter values calculated by said filtervalue calculating unit by using the feature quantity detected by saidfeature quantity detecting unit and the directional correlation valuecalculated by said directional correlation value calculating unit, andfor multiplying a result of said weighting arithmetic operations by asensitization magnification to create pixel signals of a correctlyexposed image.
 2. The image processing device according to claim 1,wherein said image processing device includes an amplitude levelcalculating unit for calculating an amplitude level of the aimed pixelwith respect to the neighboring pixels from the feature quantitydetected by the feature quantity detecting unit, and the correctlyexposed image creating unit performs the weighting arithmetic operationson the plurality of filter values calculated by the filter valuecalculating unit by using the feature quantity detected by said featurequantity detecting unit, the directional correlation value calculated bythe directional correlation value calculating unit and the amplitudelevel calculated by said amplitude level calculating unit.
 3. The imageprocessing device according to claim 1, wherein the filter valuecalculating unit calculates a low pass filter value in a one-dimensionaldirection including the aimed pixel, a median filter value in aone-dimensional direction including said aimed pixel, a two-dimensionallow pass filter value including said aimed pixel, and a two-dimensionalmedian filter value including said aimed pixel.
 4. The image processingdevice according to claim 1, wherein the feature quantity detecting unitcalculates horizontal and vertical secondary differential values of theaimed pixel as the feature quantity of the object to be photographed. 5.An image processing method comprising: a feature quantity detecting stepof a feature quantity detecting unit detecting a feature quantity of anobject to be photographed which exists around an aimed pixel in atwo-dimensional image; a directional correlation value calculating stepof a directional correlation value calculating unit calculating adirectional correlation value of the object to be photographed whichexists around said aimed pixel from the feature quantity detected bysaid feature quantity detecting unit; a filter value calculating step ofa filter value calculating unit referring to a pixel signal of saidaimed pixel and pixel signals of neighboring pixels in a neighborhood ofsaid aimed pixel to calculate a plurality of filter values; and acorrectly exposed image creating step of a correctly exposed imagecreating unit performing weighting arithmetic operations on theplurality of filter values calculated by said filter value calculatingunit by using the feature quantity detected by said feature quantitydetecting unit and the directional correlation value calculated by saiddirectional correlation value calculating unit, and for multiplying aresult of said weighting arithmetic operations by a sensitizationmagnification to create pixel signals of a correctly exposed image. 6.The image processing method according to claim 5, wherein said imageprocessing method includes an amplitude level calculating step of anamplitude level calculating unit calculating an amplitude level of theaimed pixel with respect to the neighboring pixels from the featurequantity detected by the feature quantity detecting unit, and thecorrectly exposed image creating unit performs the weighting arithmeticoperations on the plurality of filter values calculated in the filtervalue calculating step by using the feature quantity detected by saidfeature quantity detecting unit, the directional correlation valuecalculated by the directional correlation value calculating unit and theamplitude level calculated by said amplitude level calculating unit.